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People’s  everyday  experiences  are  multisensory.  For  example,  while  eating  breakfast,  we 
both  see  and  grasp  our  coffee  cup.  Moreover,  multisensory  perception  is  critical  in  some  highly 
important  situations,  as  when  a  TSA  agent  searches  a  passenger’s  bag  by  both  looking  inside  the 
bag  and  touching  its  contents  or  when  a  police  officer  frisks  a  person  using  both  visual  and 
tactile  inspection. 

In  brief,  our  research  program  uses  experimental  and  computational  methodologies  to 
study  how  people  acquire  multisensory  representations  and  how  the  use  of  these  representations 
influences  perceptual  judgements  and  decision  making.  The  program  focuses  on  people’s 
perfonnances  in  visual-haptic  and  visual-auditory  environments.  People  are  extraordinarily  good 
at  learning  infonnation  via  one  sensory  modality  but  applying  the  acquired  knowledge  when 
environments  are  perceived  via  a  different  modality,  a  phenomenon  known  as  cross-modal 
transfer.  To  date,  artificial  intelligence  systems  are  relatively  poor  at  cross-modal  transfer  when 
environments  are  defined  in  a  realistic  manner. 

People  learn  modality-independent,  conceptual  representations  from  modality-specific 
sensory  signals.  We  hypothesize  that  any  system  that  accomplishes  this  feat  will  include  three 
components:  (i)  a  representational  language  for  characterizing  modality-independent 
representations,  (ii)  a  set  of  sensory-specific  forward  models  for  mapping  from  modality- 
independent  representations  to  sensory  signals,  and  (iii)  an  inference  algorithm  for  inverting 
forward  models — that  is,  an  algorithm  for  using  sensory  signals  to  infer  modality-independent 
representations.  We  have  instantiated  this  hypothesis  in  computational  systems  in  which  the 
modality-independent  representations  are  based  on  a  probabilistic  approach  (distributed 
representations  over  latent  variables)  and  in  systems  in  which  these  representations  are  based  on 
a  probabilistic/symbolic  hybrid  approach  (probabilistic  grammars).  We  have  also  collected 
experimental  data  from  people,  and  used  our  computational  models  to  account  for  the 
experimental  findings. 

Funding  from  this  grant  supported  research  reported  in  7  journal  publications  and  3 
conference  publications  (manuscripts  based  on  2  of  the  conference  publications  are  currently 
being  prepared  for  journal  submission). 
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